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ABSTRACT 

This  paper  advocates  the  introduction  of  so-called  cognitive  and  cooperative  automation  into  vehicle 
guidance  work  systems.  The  work  system  of  UAV  guidance  will  be  taken  as  an  example.  This  idea 
radically  breaks  with  the  traditional  approach  of  supervisory  control.  According  to  our  approach  the 
human  will  no  longer  be  the  only  instance  in  the  work  process  being  responsible  for  the  pursuit  of  the 
given  work  objective.  We  call  this  novel  mode  of  human- automation  interaction  cooperative  control. 
Suchlike  automation  will  take  over  a  more  active  part  and  develop  its  own  initiative  to  achieve  the  aim  of 
work  and  thereby  enhance  the  human-machine  system ’s  performance.  This  paper  provides  a 
comprehensive  description  of  the  top-level  work  system  design  incorporating  cognitive  automation,  a  brief 
outline  on  the  realisation  of  artificial  cognition  and  the  description  of  an  application  example  taken  from 
the  multi-UAV  guidance  and  mission  management  domain.  The  experiments  show  that  cognitive 
cooperation  provides  a  high  potential  for  human  performance  enhancement  on  the  level  of  cognitive  work. 


1.0  INTRODUCTION 

Vehicle  guidance  of  any  kind  has  always  been  a  very  demanding  task  for  human  operators.  Driving  a  car 
and  navigating  in  dense  traffic  is  probably  the  most  complex  sensori-motor  and  cognitive  task  an  average 
person  is  ever  exposed  to  in  daily  life.  In  fact,  a  whole  set  of  psycho-physiological  limitations  confine 
human  performance,  e.g.  associated  with  visual  perception  or  reaction  time.  Fatigue  might  as  well  be  a 
major  issue.  Obviously  operating  a  military  aircraft  requires  much  more  specialised  human  capabilities 
and  poses  extremely  high  demands  on  pilots.  In  addition  to  the  aforementioned  problems  physiological 
human  performance  limitations  in  connection  with  e.g.  acceleration  tolerance,  hypoxia  or  motion  sickness 
arise.  With  the  advent  of  more  and  more  automation  in  the  work  environment  the  nature  of  work  has  been 
changed  considerably.  Many  of  the  purely  manual  control  tasks  vanished,  while  supervisory  control  [1] 
became  the  predominant  mode  of  human-machine  interaction  in  vehicle  guidance.  This  led  to  a  great  relief 
from  demanding  high-bandwidth  tasks  and  the  size  of  operating  crews  could  be  considerably  scaled  down, 
e.g.  the  flight  deck  crew  of  civil  airliners  from  four  or  even  five  persons  in  the  earlier  years  to  two  pilots  as 
usual  today.  Accompanied  with  that,  however,  work  environments  became  much  more  complex,  requiring 
increasing  skills  and  knowledge  on  behalf  of  the  human  operators  in  order  to  still  cope  with  the 
complexity.  In  fact,  human  cognitive  performance  limits  became  the  predominant  restriction.  Unless  high 
efforts  invested  in  the  optimisation  of  human-machine  systems  typical  reactive  designs  could  not  really 
succeed  in  counteracting  suchlike  problems.  Cognitive  and  cooperative  automation  represent  novel 
approaches  to  this  problem,  radically  breaking  with  the  tradition  of  supervisory  controlled  automation.  The 
Institute  of  Flight  Systems  (and  it’s  predecessor  organisations)  at  the  Universitat  der  Bundeswehr 
Munchen  (Munich  University  of  the  German  Armed  Forces)  has  been  working  on  experimental  prototypes 
since  more  than  two  decades  now.  Many  experimental  works  proving  the  concepts  have  been  performed 
enhancing  human-system  performance  in  particular  in  aerial  applications. 
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In  order  to  provide  a  closer  insight  into  the  approach  of  cognitive  and  cooperative  automation  in  the 
following  chapters  the  work  system  will  be  introduced  as  human  factors  engineering  theory  and  applied  to 
the  field  of  multi-UAV  guidance.  The  work  system  allows  a  systematic  systems  engineering  approach  of 
deriving  requirements  for  so-called  Artificial  Cognitive  Units  (ACUs).  In  the  next  chapter  a  brief 
description  will  be  given,  how  an  ACU  shall  be  built,  in  order  to  be  enabled  to  yield  knowledge-based 
behaviour.  Finally,  in  the  following  chapter  an  example  application  of  multi-UAV  guidance  and  mission 
management  will  be  described  and  respective  experimental  results  will  be  given. 


2.0  WORK  PROCESS  ANALYSIS  OF  HUMAN-MACHINE  SYSTEMS 
2.1  Work  Process  and  Work  System 

To  figure  out  a  solution  to  the  challenges  of  future  vehicle  guidance  systems,  the  first  step  shall  be  the 
characterisation  of  today  available  conventional  automation  in  the  work  process  as  opposed  to  the 
introduction  of  cognitive  automation.  Therefore,  the  consideration  of  the  work  system  as  top-level  human 
factors  engineering  framework  shall  be  used.  The  work  system  (see  Figure  1)  as  a  general  ergonomics 
concept  has  been  utilised  in  a  modified  definition,  adapted  to  the  application  domain  of  human-machine 
cooperation  in  flight  guidance  by  [2]  and  to  machine-machine  cooperation  by  [3]. 
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Figure  1 :  Work  system 


The  work  system  is  defined  by  the  work  objective,  being  the  main  input  into  the  process  of  work.  The 
work  objective  mostly  comes  as  an  instruction,  order  or  command  from  a  different  supervising  agency 
with  its  own  work  processes.  Further  constraining  factors  to  the  work  process  are  environmental 
conditions  and  supplies.  At  its  output,  the  work  system  provides  the  work  result  including  the  current  state 
of  work  and  what  has  been  accomplished  by  the  work  process  as  a  physical  change  of  the  real  world  [4]. 

The  work  system  itself  consists  of  two  major  elements,  i.e.  the  operating  force  and  the  operation¬ 
supporting  means,  as  characterised  in  some  more  detail  in  the  following: 

•  The  operating  force  is  the  high  end  decision  component  of  the  work  system.  It  is  the  only 
component  which  pursues  the  complete  work  objective.  It  determines  and  supervises  what  will 
happen  in  the  course  of  the  work  process  and  which  operation-supporting  means  will  be  deployed 
at  what  time.  The  operating  force  is  the  work  system  component  with  the  highest  authority  level. 
One  major  characteristic  of  especially  a  human  representing  the  operating  force  is  the  capability 
of  defining  the  work  objective  by  himself  (see  Figure  1).  Besides  operating  on  the  basis  of  full 
authority  competence  this  is  the  decisive  criterion  for  what  we  call  an  autonomous  system  (see 
also  discussion  below). 
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•  The  concept  of  operation- supporting  means  can  be  seen  as  a  container  for  whatever  artefacts  are 
available  today  to  make  use  of  in  the  work  process,  including  basic  work  site  settings,  non- 
powered  tools  and  machines.  The  latter  might  be  a  vehicle  in  the  case  of  a  transport  work  process, 
but  also  computerised  devices  of  automation.  A  robotic  system  in  this  sense  could  be  seen  as 
highly  automated  machine  as  part  of  the  operation-supporting  means  of  a  work  system  as  well.  In 
the  application  domain  of  flight  guidance  currently  used  auto-flight  or  autopilot  systems  including 
the  human-machine  control  interface  can  serve  as  typical  examples  for  automation.  Common  to 
the  nature  of  various  operation-supporting  means  is  the  fact  that  they  facilitate  the  performance  of 
certain  sub-tasks,  and  only  that.  By  nature,  such  a  sub-task  does  not  form  a  work  system  by  itself, 
obviously  being  only  one  part  of  another  higher  level  work  task.  According  to  the  common 
ergonomic  design  philosophy,  mostly  the  operation-supporting  means  are  subjected  to  the 
endeavours  of  optimisation  in  order  to  achieve  overall  system  requirements  and  accomplish 
further  improvements. 

As  mentioned  earlier,  these  elements  will  be  combined  within  the  work  system  in  order  to  achieve  a 
certain  work  result  on  the  basis  of  a  given  work  objective.  The  accomplishment  of  a  flight  mission  (i.e.  the 
work  objective)  may  give  a  good  idea  of  what  is  meant  here.  In  this  case,  the  work  system  will  consist  of 
an  air-crew  being  the  operating  force,  and  the  aircraft  including  its  automated  on-board  functions  as  well 
as  any  required  infrastructure  representing  the  operation-supporting  means. 

The  replacement  of  human  work  by  continuously  expanding  automated  functions,  as  it  could  be  observed 
throughout  the  whole  history  of  industrial  mechanisation  and  automation,  leads  to  technical  solutions  of 
steadily  increased  authority.  Simultaneously,  the  human  operator  is  continuously  being  pushed  further  into 
the  role  of  supervising  more  and  more  machines  and,  at  the  same  time,  more  and  more  complex  ones.  The 
process  of  substituting  the  responsibilities  of  the  human  in  a  work  system  might  be  driven  up  to  a  degree 
where  the  operator’s  capabilities  would  be  completely  substituted  by  automated  functions.  From  a  purely 
technology  minded  standpoint,  the  human  could  theoretically  be  dropped  out  of  the  work  system  as  a 
consequence.  In  case  that  the  automation  is  just  like  the  human  capable  and  allowed  to  self-assign  a  work 
objective,  the  resulting  artefact  could  be  called  an  artificial  autonomous  system  (see  Figure  2). 


Figure  2:  Hypothetical  conversion  of  an  ordinary  autonomous  work  system  with  human  operator 

into  an  artificial  autonomous  system 


However,  considering  such  an  artificial  autonomous  system,  there  are  two  good  reasons,  why  it  is  not 
desirable  to  create  a  machine  like  this: 

•  From  an  ethical  point  of  view  we  want  to  refuse  building  machines,  which  potentially  could  self¬ 
assign  a  work  objective,  as  long  as  this  implies  the  possibility  of  unforeseeable,  harmful 
consequences  for  us  humans. 
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•  From  a  pragmatic  point  of  view  we  do  not  need  machines  not  being  subjected  to  human  authority. 
Such  technological  artefacts  for  themselves  are  of  no  use,  since  they  are  no  longer  serving  the 
human  for  his  work  in  its  broadest  sense. 

Still,  technology  may  provide  very  powerful  automation  capabilities  replacing  human  performance  if 
useful,  unless  it  is  not  entitled  to  the  authority  of  self-assigning  a  work  objective.  We  call  the  resulting 
artefact  a(n)  (artificial)  semi-autonomous  system,  the  structure  of  which  is  very  similar  to  that  of  a  work 
system.  Figure  3  shows  such  a  semi-autonomous  system  which  is  always  part  of  the  operation-supporting 
means  of  a  superior  work  system.  In  this  set-up,  this  operation-supporting  semi-autonomous  system  will 
not  be  allowed  to  self-define  an  objective.  It  will  receive  it  from  the  operating  force  of  the  associated  work 
system  which  at  least  comprises  one  human  operator.  Surely,  such  a  semi-autonomous  system  may  work 
very  independently,  without  continuous  guidance  by  the  operating  force. 


Figure  3:  Semi-autonomous  system  as  part  of  a  work  system 

In  conclusion,  the  definition  of  a  work  system  excludes  the  existence  of  a  purely  artificial  one.  There  will 
be  no  artificial  autonomous  system  addressed  as  a  work  system.  There  will  be  no  artificial  system 
addressed  which  would  have  got  the  capacity  to  generate  a  work  objective  on  its  own.  Instead,  we  claim 
the  necessary  condition  that  there  will  be  at  least  one  human  operator  as  part  of  the  operating  force  of  the 
work  system.  In  other  words,  every  technological  artefact,  no  matter  how  high  its  “level  of  autonomy” 
might  be  advertised  by  its  creators,  will  finally  either  support  human  work  (as  part  of  a  work  system)  in  its 
broadest  sense  or  not  be  of  any  use  at  all. 

2.2  Experiences  with  complex  automation 

The  concept  of  having  conventionally  automated  functions  in  the  work  process  as  low  authority  artificial 
specialists  being  good  for  a  particular  execution  support  but  lacking  the  full  perspective,  was  very 
convenient  as  long  as  only  simple  automated  functions  were  used  for  simple  tasks.  It  was  of  no  harm  that 
the  responsibility  load  not  to  violate  the  prime  goals  of  the  work  process  was  exclusively  on  the  human 
operator’s  shoulders.  However,  in  highly  automated  work  systems  for  complex  and  demanding  work 
objectives  as  we  are  used  to  now,  like,  for  instance,  a  demanding  mission  of  a  modern  fighter  aircraft,  the 
situation  is  different.  Here,  conventionally  automated  functions  with  a  high  degree  of  complexity  are 
accommodated  as  operation-supporting  functions.  They  are  considered  as  necessary  to  keep  the  human 
operator  freed  from  a  sufficiently  high  number  of  tasks  which  otherwise  could  lead  to  overloads.  This 
might  become  critical,  though,  since  with  increasing  conventional  automation  complexity,  the  task  of  the 
operator  as  supervisor  becomes  more  and  more  complex,  too.  Instead  of  the  intended  unloading,  danger  of 
overcharge  of  the  operator  may  inevitably  arise  in  certain  situations,  resulting  in  a  loss  of  situation 
awareness  and  control  performance,  i.e.  lacking  to  comply  with  the  prime  goals.  The  operator  might  be 
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unaware  of  discrepancies  between  sub-task  activities  of  automated  functions  or/and  the  prime  goal 
necessities  or  even  of  his  insufficiency  to  adapt  to  this  inadequacy.  This  is  just  the  opposite  of  what  was 
intended  by  the  introduction  of  automation.  [4] 

This  issue  fits  well  the  characteristics  of  a  vicious  circle.  With  respect  to  the  extension  of  conventional 
automation,  such  a  vicious  circle  is  depicted  in  Figure  4  for  the  application  field  of  UAV  guidance  and 
control. 


increasing  human 
system  monitoring 
tasks 


human 

error 


extended 

automation 


higher 

compiexity 


Figure  4:  The  vicious  circie  of  extending  conventionai  automation 


As  characteristic  of  reactive  designs  to  inescapably  occurring  human  error  in  human  supervised 
automation  systems,  the  extension  of  automation  is  taken  as  solution.  Designers  transfer  the  authority  over 
more  and  more  comprehensive  work  process  tasks  from  the  operating  force  i.e.  the  human  operator  to 
increasingly  complex  automated  functions  being  part  of  the  operation-supporting  means.  Although,  being 
relieved  from  workload  in  the  first  place,  the  human  operator  will  be  burdened  with  extra  demands  from 
increasing  system  monitoring  and  management  tasks.  At  a  certain  point  the  human  operator  cannot  but  fail 
in  the  supervision  of  suchlike  automated  systems.  In  reaction,  as  being  used  to,  typically  the  only  solution 
considered  is  to  further  extend  the  share  of  automation  as  part  of  the  operation-supporting  means  in 
conjunction  with  further  increase  of  complexity.  In  conclusion,  as  we  are  interested  in  a  design  with  gains 
in  productivity  of  a  work  system,  we  experience  that  the  increase  of  system  complexity  due  to  further 
addition  of  conventional  automation  results  in  less  and  less  increase  of  productivity  gain  and  might 
eventually  even  end  up  with  decreases. 

The  deficiencies  of  conventional  automation  finally  leading  to  the  vicious  circle  described  have  been 
experienced  for  quite  a  time  now  (see  [5])  and  are  well-captured  in  explicit  terms  in  the  meantime  by  the 
phenomena  known  under  e.g.  automation  brittleness,  opacity  (entailing  increased  demands  on  monitoring 
the  automation  status),  literalism,  clumsiness,  and  data  overload,  all  of  them  being  consequences  of  too 
high  system  complexity. 

2.3  Dual-Mode  Cognitive  Automation 

Traditionally,  a  human  or  a  human  team  (cf.  e.g.  [2])  represents  the  operating  force  in  the  work  system.  In 
the  conventional  sense  the  human  operating  force  provides  the  capability  of  cognition  within  a  work 
system,  whereas  the  operation-supporting  means  do  not.  In  order  to  overcome  known  shortcomings  of 
such  conventional  automation  (e.g.  [5] [6])  which  is  oftentimes  by  far  too  complex  to  be  handled  properly, 
a  configuration  of  the  work  system  is  suggested,  where  so-called  Artificial  Cognitive  Units  (ACU)  are 
introduced. 
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Inserting  ACUs  into  the  work  system  as  opposed  to  the  further  enhancement  of  conventionally  automated 
functions  or  the  addition  of  further  humans  (first  row  in  Figure  5)  adds  a  new  level  of  automation,  i.e.  the 
cognitive  level  to  the  work  system  (second  row  in  Figure  5).  The  possibility  to  shape  an  ACU  being  either 
part  of  the  operating  force  {operating  ACU)  or  being  part  of  the  operation-supporting  means  {supporting 
ACU)  defines  two  modes  of  cognitive  automation  [4].  Both  modes  might  be  combined  together  with 
conventional  automation  within  one  work  system. 


Figure  5:  Introducing  Artificial  Cognitive  Units  (ACUs)  into  the  work  system 

Common  to  these  two  modes  of  automation  is  that  they  incorporate  artificial  cognition.  Onken  [2] 
describes  the  nature  of  suchlike  cognitive  automation  as  follows: 

'As  opposed  to  conventional  automation,  cognitive  automation  works  on  the  basis  of  comprehensive 
knowledge  about  the  work  process  objectives  and  goals  [...],  pertinent  task  options  and  necessary  data 
describing  the  current  situation  in  the  work  process.  Therefore,  cognitive  automation  is  prime-goal- 
oriented.  [2] 

Particularly  due  to  the  orientation  of  its  behaviour  towards  explicitly  represented  goals  and  its 
understanding  of  the  current  situation,  cognitive  automation  has  the  potential  to  avoid  problems  of 
conventional  approaches  such  as  complexity  and  brittleness  which  have  been  thoroughly  investigated  by 
Billings  [5]  and  others.  A  detailed  explanation  of  this  effect  can  be  found  in  [4]. 

Concerning  the  application  of  cognitive  automation  as  operating  ACU,  i.e.  on  the  right  hand  side  of  Figure 
5,  where  the  human  operator  and  the  ACU  form  the  operating  force  as  a  team  [2]  comments  that  in  this 
configuration  the  ACU  has  reached  'f...]  the  high-end  authority  level  for  decisions  in  the  work  system, 
which  was,  so  far,  occupied  by  the  human  operator  alone. [2] 

As  a  consequence  of  this  consideration  both  team  members  -  human  operator  and  ACU  -  have  to  have  the 
obligation  to  apply  their  specific  capabilities,  which  might  be  overlapping,  in  order  to  pursue  the  overall 
work  objective  best.  Hence,  an  operating  ACU  is  always  characterised  by  the  incorporation  of  the 
functionality  of  what  we  call  an  assistant  system.  Such  an  assistant  system  can  be  a  supplement  to  the 
operating  force,  i.e.  to  the  human  operator  or  the  human  team.  In  this  case  it  is  mostly  of  virtual  nature 
providing  informational  assistance.  On  the  other  hand  it  can  be  embodied  in  an  independent  machine  (e.g. 
robot,  UAV)  substituting  an  otherwise  necessary  additional  human  operator.  In  this  case  the  assistance  is 
of  physical  nature  in  addition.  The  design  of  the  corresponding  operating  force  of  a  work  system. 
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especially  the  design  of  the  assistant  system  (i.e.  an  operating  ACU)  can  be  specified  by  founding  it  on  the 
compliance  with  the  following  basic  requirements  (see  also  [7]  and  [4]): 

Requirement  1: 

The  assistant  system  has  to  be  able  to  present  the  full  picture  of  the  work  situation  from  its  own 
perspective  and  has  to  do  its  best  by  own  initiatives  to  ensure  that  the  attention  of  the  assisted  human 
operator(s)  is  placed  with  priority  on  the  objectively  most  urgent  task  or  subtask. 

Requirement  1  leads  to  functions  that  become  active  only  at  certain,  normally  rare  occasions  for 
contacting  the  human  operator(s)  with  advisory  messages  when  overtaxing  of  the  human  operator(s)  seems 
possible  in  the  task  category  of  situation  assessment  and  interpretation.  This  takes  place  in  a  way  such  that 
the  human  operator(s)  can  make  up  his  (her/their)  mind  to  accept  the  advice  or  not.  This  function  is 
entirely  advisory. 

Requirement  2: 

If  according  to  requirement  1  the  assistant  system  can  securely  identify  as  part  of  the  situation 
interpretation  that  the  human  operator(s)  cannot  carry  out  the  objectively  most  urgent  task  because  of 
overtaxing,  then  the  assistant  system  has  to  do  its  best  by  own  initiatives  to  automatically  transfer  this 
situation  into  another  one  which  can  be  handled  normally  by  the  assisted  human  operator(s). 

Since  requirement  2  is  based  on  sufficient  fulfilment  of  requirement  1,  the  technical  realisation  of  the 
assisting  function  according  to  requirement  2  consists  of  two  elements  (1)  to  realise  reliable  identification 
of  situations  with  overtaxing  of  the  human  operator,  albeit  he  or  she  has  got  full  situation  awareness,  and 
(2)  to  realise  reliable  automated  functionality  to  overcome  this  situation.  At  this  point,  the  assistant  system 
is  temporarily  replacing  a  human  team  member.  In  essence,  the  corresponding  functions  are  directly 
impinging  on  the  work  product  by  automatically  carrying  out  the  due  tasks  which  otherwise  the  human 
operator  was  supposed  to  do.  Depending  on  the  situation,  explaining  information  might  be  given  to  the 
human  operator  that  he  is  enabled  to  take  over  again  as  soon  as  possible.  In  the  context  of  describing  the 
essence  of  temporarily  substituting  assistance,  this  problem  is  being  tackled  by  the  work  on  adaptive 
automation  (see  e.g.  [8]  and  [9]  for  own  works  in  the  field) 

Requirement  3: 

If  there  are  cognitive  tasks,  the  human  operator(s)  is(are)  principally  not  capable  to  accomplish,  or  which 
are  of  too  high  a  risk  or  likely  a  cause  of  too  high  costs,  these  tasks  are  to  be  allocated  to  the  assistant 
system  or  operation-supporting  means,  possibly  a  supporting  cognitive  unit. 

Requirement  3  refers  to  an  a-priori  design  decision  to  transfer  certain  cognitive  tasks  within  a  given  work 
context  to  the  assistant  system  permanently.  This  may  include  that  the  assistant  system  decides  to  utilise 
certain  operation-supporting  means  in  order  to  fulfil  the  task.  In  this  case  the  assistant  system  can  have  its 
own  exclusive  operation-supporting  means  at  its  disposal.  Whereas  assistant  functions  derived  from 
requirement  1  or  2  most  typically  will  be  carried  out  at  the  operator’s  work  station,  assistant  functions 
derived  from  requirement  3  might  as  well  be  dislocated,  e.g.  aboard  a  UAV.  In  this  case  the  operating 
ACU  is  guiding  the  UAV  at  a  high  degree  of  autonomy  forming  the  operating  force  with  the  human 
located  somewhere  else. 

The  introduction  of  an  operating  ACU  is  the  basis  for  human-machine  symbiosis  in  its  real  sense  and 
guides  the  way  out  of  the  vicious  circle  (see  Figure  4)  as  shown  in  Figure  6.  Although  cognitive 
automation  does  not  break  the  general  trend  of  raising  complexity  as  such,  the  introduction  of  the 
cooperative  control  mode  counteracts  typical  problems  with  supervisory  control  of  increasingly  complex 
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automation  by  definitely  mitigating  the  system  monitoring  and  management  task  load  through  human - 
automation  cooperation. 


human 


Figure  6:  Escaping  the  vicious  circie  of  progressing  automation 


2.4  Introduction  of  Cognitive  Automation  in  UAV  Guidance  Work  System 

After  having  discussed  the  introduction  of  cognitive  and  cooperative  automation  into  work  systems  in 
general  this  section  will  pick  up  the  UAV  guidance  work  system  as  an  example  and  discuss  several 
variations  of  it  with  respect  to  the  possibilities  of  introduction  of  cognitive  automation. 

To  begin  with,  let  us  consider  a  work  system  for  the  guidance  of  multiple  UAVs  by  a  single  operator 
without  cognitive  automation.  The  UAVs  can  be  used  according  to  their  capabilities  in  order  to  achieve 
the  work  objective.  Concerning  the  tasks  of  the  UAV  operator  (just  considering  UAV  flight  guidance),  he 
usually  has  to  plan  and  supervise  mission  plans  in  parallel  processes  for  several  UAVs.  Moreover,  he  has 
to  decide,  how  his  work  objective  (e.g.  area  reconnaissance)  can  be  split  into  several  sub-tasks,  which 
UAV  shall  complete  which  sub-tasks  (e.g.  certain  sector  of  area  of  interest)  and  how  this  sub-task  can  be 
executed  (e.g.  definition  of  a  search  pattern).  The  execution  of  the  task  is  followed  by  the  fusion  of  the 
results,  as  the  operator  is  the  only  instance  within  the  work  system,  which  has  overview  of  relevant  sub¬ 
tasks  and  their  dependencies. 

In  order  to  support  the  operator  within  this  work  system,  additional,  cognitive  automation  can  be 
introduced  in  different  ways.  In  a  first  step,  the  UAVs  can  be  promoted  to  become  semi-autonomous 
systems,  which  are  operated  by  a  supporting  ACU  (see  Figure  7).  This  supporting  ACU  can  access 
necessary  information  and  can  control  the  UAV  by  the  use  of  various  sub-systems  or  communicate  with 
the  operator  by  data  link. 

This  configuration  does  not  require  detailed  instructions  for  the  UAVs  with  respect  to  how  a  certain  task 
shall  be  executed  (e.g.  specification  of  a  mission  plan),  but  the  supporting  ACUs  are  capable  of 
understanding  tasks  on  the  abstraction  level  of  a  mission  order  and  executing  them  by  the  use  of  available 
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operation-supporting  means.  Thus,  the  UAVs  get  more  independent  from  highly  frequent  and  detailed 
instructions  from  the  operator  so  that  there  are  less  critical  requirements  concerning  data  link  availability 
and  time  critical  reactions  of  the  operator.  Moreover,  the  operator  is  unburdened  from  operation  of  UAV 
sub-systems,  although  he  might  have  the  possibility  to  access  them. 

In  this  configuration,  the  semi-autonomous  systems  are  acting  independently  from  each  other  (see  Figure 
7).  Thus,  the  operator  has  to  care  for  appropriate  allocation  of  tasks  to  UAVs  as  well  as  re-allocation  in 
case  of  relevant  changes  in  the  situation  and  supervision  of  task  execution  by  the  supporting  ACUs. 


Figure  7:  Independent  supporting  ACUs  within  UAV  guidance  work  system 

The  next  step  combines  the  semi-autonomous  system  to  one  dislocated  semi-autonomous  system 
consisting  of  a  team  of  supporting  ACUs.  Hereby,  each  supporting  ACU  is  associated  with  one  UAV  and 
located  onboard  this  UAV  (cf.  Figure  8). 


Figure  8:  Team  of  supporting  ACUs 

This  configuration  facilitates  the  allocation  of  one  task  to  a  team  of  UAVs,  that  determines  necessary  sub¬ 
tasks  as  well  as  an  appropriate  task  allocation  by  itself.  Usually,  the  team  will  account  for  situation 
changes  and  adapt  if  required.  While  so  far  the  structure  of  the  work  system  changed  in  case  a  UAV  was 
lost  or  the  number  of  UAVs  increased,  the  operator  now  has  a  operation-supporting  means  at  hand,  which 
is  much  more  complex  than  a  semi-autonomous  system  consisting  of  one  UAV,  but  which  unburdens  him 
from  various  tasks  and  tries  to  maintain  situation  awareness  with  respect  to  the  collaboration  of  UAVs. 

Such  a  semi-autonomous  UAV  team  is  capable  of  executing  a  given  task  usually  independent  from  human 
intervention,  but  cannot  contribute  to  the  work  objective  of  the  overall  work  system  as  it  is  not  known 
within  the  team.  In  case  such  a  capability  is  required,  the  ACUs  have  to  take  the  shape  of  operating  ACUs 
(see  Figure  9).  This  step  changes  the  interaction  of  human  operator  and  ACUs  fundamentally  as  the  ACUs 
are  no  longer  supervised  by  the  human  in  a  hierarchical  manner  but  now  form  a  team  within  the  operating 
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force  which  cooperatively  tries  to  achieve  the  work  objective.  Thus,  the  operating  ACUs  in  this  setup  have 
to  be  capable  of  human-machine  cooperation. 


Figure  9:  Team  of  operating  ACUs  and  human  operator 


Like  previously,  also  this  configuration  comprises  several  ACUs  because  still  each  ACU  is  associated 
with  one  UAV  and  located  onboard  in  order  to  be  able  to  compensate  for  temporary  data  link  loss  and  be 
robust  with  respect  to  the  loss  of  a  single  UAV.  Here,  a  special  characteristic  of  the  operating  force  is 
used,  namely  that  its  elements  may  be  dislocated  [4].  Moreover,  from  the  perspective  of  a  work  system 
designer,  this  set-up  follows  the  recommendation  to  introduce  as  much  automation  as  possible  into  the 
operating  force  [4]  as  it  is  assumed  that  the  quality  of  decisions  and  the  performance  of  the  work  system  as 
a  whole  is  the  better  the  more  automation  knows,  understands  and  pursues  the  work  objective. 

The  difference  between  a  team  of  supporting  ACUs  and  a  team  of  operating  ACUs  each  of  them  guiding  a 
dedicated  UAV  shall  be  clarified  using  an  example  from  the  MUM-T  domain.  Here,  a  manned  helicopter 
has  several  UAVs  guided  by  the  aforementioned  ACUs  at  its  disposal  for  route  reconnaissance.  This 
manned  helicopter  in  on  the  way  to  a  certain  site  and  needs  information  concerning  the  planned  route. 
Moreover,  there  are  several  alternative  routes  briefed.  In  case  the  UAVs  are  guided  by  a  team  of 
supporting  ACUs,  the  operator  can  task  them  to  reconnoitre  a  certain  route,  but  has  to  care  for  re -tasking 
in  case  other  routes  shall  be  explored  because  the  ACUs  do  not  know  that  the  manned  helicopter  is  on  the 
way  to  a  certain  location.  In  contrast,  if  the  UAVs  are  guided  by  a  team  of  operating  ACUs,  the  latter  do 
know  the  work  objective  and  can  hence  propose  or  execute  the  exploration  of  alternative  routes  by 
themselves  in  case  of  situation  changes. 


Figure  10:  Team  of  supporting  ACUs  and  operating  ACU  as  operator  assistant  system 
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In  order  to  further  optimise  the  UAV  guidance  work  system  that  has  been  considered  so  far,  an 
(additional)  operating  ACU  should  be  introduced  into  each  configuration  [4].  This  is  especially  necessary 
for  set-ups  which  do  not  contain  an  operating  ACU  yet  in  order  to  counteract  problems  which  can  be 
expected  with  increasing  complexity  of  automation  [5].  The  new  operating  ACU  behaves  as  an  assistant  to 
the  human  operator,  i.e.  it  supports  him  in  the  execution  of  his  specific  tasks.  Concerning  the  supervision 
of  a  team  of  supporting  ACUs  (cf.  Figure  10),  such  an  assistant  system  would  need  concepts  of  team  work 
and  appropriate  task  distribution  among  the  team  members  in  order  to  be  able  to  derive  the  need  for  action. 
Moreover,  knowledge  about  various  alternatives  to  intervene  is  necessary  as  well  as  an  understanding  of 
the  operator’s  current  workload  and  attention  focus  in  order  to  provide  situation  adapted  support. 


3.0  BUILDING  ARTIFICIAL  COGNITIVE  UNITS 

As  mentioned  above,  we  introduce  so-called  Artificial  Cognitive  Units  (ACUs)  aboard  UAVs  which  are 
capable  of  goal-directed  planning  of  their  behaviour  while  considering  the  current  situation.  In  order  to 
perform  well  in  as  many  situational  configurations  as  possible  these  ACUs  have  to  be  able  to  exhibit 
behaviour  on  all  levels  of  human  performance  as  introduced  by  [10]: 

•  Skill-based  behaviour  is  characterised  by  highly  automated  and  efficient  execution  of  sensori¬ 
motor  patterns  without  the  need  to  be  aware  of.  Just  like  an  experienced  helicopter  pilot  can 
perform  a  hovering  task  on  this  level,  a  controller  stabilising  an  airborne  platform  would  exhibit 
the  equivalent  of  skill-based  behaviour. 

•  Rule-based  behaviour  in  contrast  requires  attentional  resources  from  humans  and  can  be  observed 
in  standard  task  situations.  Then,  a  direct  situation-task-mapping  takes  place  and  the  appropriate 
tasks  can  be  executed  by  means  of  skill-based  capabilities.  Typical  rule-based  behaviour  in  the 
aviation  domain  can  be  observed  when  processing  check  lists  e.g.  before  departure. 

•  Knowledge-based  behaviour  is  of  importance  in  situations,  which  have  not  been  experienced 
before  and  for  which  it  is  not  known  what  should  be  done  next.  For  example,  a  pilot  might  not 
immediately  have  an  appropriate  solution  to  a  situation  in  which  certain  mission  relevant  tasks 
have  to  be  completed  but  at  the  same  time  unexpected  events  such  as  a  change  of  the  tactical 
situation  or  onboard  available  resources  occur.  In  order  to  determine  the  next  steps,  at  first,  the 
situation  has  to  be  understood,  then,  currently  relevant  goals  will  be  determined  and  finally, 
appropriate  actions  have  to  be  planned  which  are  suited  to  achieve  the  desired  state. 

While  skill-based  and  rule-based  behaviour  can  be  implemented  into  technical  systems  quite 
straightforward,  performance  on  the  knowledge-based  level  is  much  harder  to  realise,  because  developers 
have  to  enable  the  system  to  understand  the  situation,  to  reason  about  super  ordinate  goals,  to  decide  what 
to  achieve  next  and  to  plan  appropriate  action  sequences. 

A  paradigm  for  the  design  for  artificial  cognitive  systems  with  such  capabilities  is  the  Cognitive  Process 
(CP)  [4] [11] [12],  which  is  depicted  in  Figure  11.  It  is  a  model  of  human  information  processing  and 
describes  a-priori  knowledge  models  necessary  for  the  implementation  of  especially  knowledge -based 
behaviour  as  well  as  transformation  steps  actually  processing  the  knowledge. 

The  transformer  interpretation  uses  environment  models  to  gather  an  understanding  of  the  current  situation 
on  the  basis  of  input  data  from  the  environment.  This  belief  is  the  most  important  input  for  the 
determination  of  currently  relevant  goals  to  be  achieved  next.  These  are  derived  from  desires,  which 
describe  all  goals  that  can  potentially  be  prosecuted  by  the  ACU.  The  transformer  planning  assembles 
available  action  alternatives  to  a  plan,  which  is  suited  to  achieve  the  goals.  Finally,  the  plan  is  being 
executed  and  instructions  are  sent  to  the  output  interface. 
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For  the  realisation  of  ACUs  based  on  this  paradigm,  the  cognitive  system  architecture  COS  A  [11]  has 
been  developed  which  provides  a  framework  implementing  application-independent  parts  of  the  CP,  i.e. 
knowledge  processing  by  the  transformers.  Moreover,  the  development  of  application-specific  a-priori 
knowledge  is  supported  by  the  Cognitive  Programming  Language  CPL,  which  allows  to  actually 
formulate  environment  models,  desires,  action  alternatives  and  instruction  models.  [13] 


Figure  11 :  The  Cognitive  Process  for  generating  knowiedge-based  behaviour  in  an  ACU 


4.0  APPLICATION  OF  COGNITIVE  AND  COOPERATIVE  AUTOMATION 

At  the  Universitat  der  Bundeswehr  (UBM)  in  Munich,  Artificial  Cognitive  Units  have  been  and  are  being 
developed  in  flight  guidance  both  as  assistant  systems  and  onboard  mission  management  systems.  Within 
the  last  years,  focus  shifted  from  manned  flight  and  pilot  assistance  to  UAV  guidance.  Topics  in  this  area 
have  been  single  UAV  mission  management  in  a  reconnaissance  mission,  predominantly  to  prove  that  the 
Cognitive  Process  and  COS  A  [11]  are  a  valid  approach  to  UAV  guidance  as  well  as  cooperative  mission 
accomplishment  by  multiple  UAVs  in  a  hostile  area  with  air-to-ground-attack  as  primary  mission  task. 

The  project  described  here  was  focused  on  the  activity  of  a  team  of  UAVs  which  is  supposed  to 
accomplish  a  given  mission  together  [14][15].  Each  of  the  UAVs  is  thereby  piloted  by  an  ACU,  which  is 
the  same  for  each  UAV.  In  the  first  place,  the  ACUs  were  to  behave  on  the  knowledge-based  level  of 
performance,  i.e.  to  process  situational  cues,  determine  goals,  and  plan  their  task  agenda.  Thus,  the  design 
of  the  a-priori  knowledge  was  deliberately  done  without  knowledge  about  task  situations  on  the  rule-based 
behavioural  level.  As  a  consequence,  all  UAV  actions  can  directly  be  tracked  back  to  goals  as  being 
determined  on  the  knowledge -based  level.  From  a  capability  point  of  view,  the  focus  of  the  project  was 
cooperative  mission  management  rather  than  ordinary  flight  planning.  The  latter  and  the  coverage  of  skill- 
based  action  control  were  assumed  to  be  given  as  well-established  conventional  automation  like  autopilot 
and  flight  management  systems. 


4.1  Scenario 

The  scenario  that  has  been  considered  consists  of  a  high  value  hostile  target  at  a  certain  location  which  has 
to  be  attacked  and  some  threats  which  have  to  be  avoided  or  suppressed  in  order  to  create  a  safe  corridor 
to  the  target  (see  Figure  12).  Some  of  the  threat  areas  are  known  before  the  mission  starts  while  others 
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may  pop  up  unexpectedly  during  the  course  of  the  mission.  One  of  the  UAV  team  members  (“Attack- 
UAV”)  is  equipped  with  the  means  to  attack  the  target.  However,  this  UAV  cannot  protect  itself  against 
threats  and  has  no  sensors  to  detect  new  threats  in  the  theatre.  The  SEAD-UAVs  in  contrast  cannot  attack 
the  target,  but  have  HARMs  with  them  in  order  to  suppress  or  destroy  SAM-sites  and  have  got  sensor 
equipment  for  on-board  detection  of  threats  as  well  as  approaching  missiles.  While  the  UAVs  are 
equipped  differently,  they  are  piloted  by  an  ACU  of  same  cognitive  capabilities  as  was  earlier  alluded  to. 


Figure  12:  Scenario  of  SEAD  /  Attack  mission 


In  order  to  accomplish  the  sketched  mission,  the  participating  UAVs  have  to  cover  the  following 
capabilities  [16]: 

•  Use  of  UAV  equipment  of  conventional  automation,  such  as  an  autopilot,  a  flight  management 
system  and  a  flight  planner  minimising  threat  exposure. 

•  Safe  flight,  i.e.  ensuring  that  the  UAV  can  fly  within  3D-space  without  colliding  with  other  UAVs 
or  terrain. 

•  Accomplishment  of  mission  tasks  such  as  “suppress  threat”. 

•  Cooperative  mission  accomplishment,  i.e.  the  ability  to  cooperate  with  other  UAVs  in  order  to 
achieve  the  common  objective,  namely  the  mission  assigned  to  the  team. 

All  of  these  aspects  had  to  be  considered  when  implementing  a  prototype  of  the  ACUs.  Since  the  focus  of 
the  project  was  on  cooperative  behaviour,  the  first  three  aspects  were  only  covered  as  much  as  needed  by 
the  cooperative  capability.  In  the  following,  the  chosen  work  system  setup  and  the  development  of  the 
prototype  with  respect  to  the  last  aspect,  i.e.  cooperation,  will  be  described  in  some  more  detail. 


4.2  Selected  Work  System  Set-Up 

The  team  of  UAVs  as  described  above  has  been  designed  to  be  the  main  operation-supporting  means  of  a 
work  system  as  depicted  in  Figure  8.  Hereby,  a  human  operator  forming  the  operating  force  has  a  semi- 
autonomous  system  consisting  of  the  UAV  team  at  hand  in  order  to  achieve  his  work  objective.  As  the 
ACUs  are  capable  of  working  together,  the  operating  force  can  task  the  team  as  a  whole  and  receives 
feedback  on  task  accomplishment  by  one  dedicated  spokesman  of  the  team.  This  spokesman  has  no 
decision  authority  higher  than  the  one  of  the  other  team  members,  but  is  responsible  for  communication 
with  the  operator. 

4.3  Design 

The  design  of  the  Artificial  Cognitive  Units  is  based  on  the  Cognitive  Process  as  presented  in  chapter  3. 
This  section  will  detail  some  of  the  knowledge  models  related  to  cooperative  behaviour.  At  first  desires 
will  be  specified,  followed  by  action  alternatives,  instruction  models,  and  finally  environment  models. 
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The  main  desires  for  producing  cooperative  behaviour  of  ACUs  are  shown  in  Figure  13.  The  first  group  of 
desires  refers  to  the  achievement  of  the  common  objective,  i.e.  usually  the  given  mission.  Therefore, 
commitments  have  to  be  managed  and  the  associated  tasks  accomplished.  Commitment  management 
hereby  refers  to  both  accepting  and  dropping  commitments  under  certain  circumstances.  Usually,  a 
commitment  is  accepted  by  an  individual  team  member  or  a  team  if  somebody  (e.g.  operator)  requests  the 
accomplishment  of  a  certain  task  and  the  accomplishment  of  this  task  is  judged  as  being  feasible.  A 
commitment  should  be  dropped  if  it  is  achieved,  irrelevant  or  unachievable  [17] [18].  As  this  does  not  only 
refer  to  own  commitments,  but  also  to  commitments  which  other  team  members  have  accepted  due  to  own 
requests,  irrelevant  dialogs  should  not  be  continued.  Moreover,  a  team  member  should  not  only  consider 
each  commitment  separately  but  should  also  consider  whether  all  its  commitments  are  achievable  as  a 
whole.  Team  members  as  individuals  are  responsible  for  the  actual  accomplishment  of  tasks  associated 
with  commitments,  therefore,  commitments  have  to  be  arranged  in  a  certain  order  in  which  they  shall  be 
considered  (so-called  ‘agenda’)  and  actually  be  complied  with  at  a  certain  point  in  time. 
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Figure  13:  Desires  for  cooperation  (see  [19]) 


Following  the  requirements  derived  by  [5]  to  be  imposed  upon  cooperative  agents,  cooperative  behaviour 
includes  appropriate  information  exchange  within  the  team  driven  by  the  desire  to  have  all  necessary 
information  about  the  other  team  members  and  to  keep  them  informed  with  respect  to  e.g.  capabilities, 
resources,  and  commitments.  The  distribution  of  tasks  within  a  team  should  be  organised  in  a  way  that  all 
tasks  which  should  be  carried  out  are  assigned  to  a  team  member  and  at  the  same  time  workload  is 
balanced  among  the  team  members.  Finally,  in  the  project  described  here,  the  team  is  structured  such  that 
all  team  members  are  peers,  but  that  communication  with  actors  not  belonging  to  the  team  is  performed  by 
a  spokesman.  Therefore,  a  desire  “have  spokesman”  has  been  formulated. 

Within  the  scenario  described  earlier  three  kinds  of  interdependencies  between  team  members  are 
relevant.  First  of  all,  shared  resources  (here  a  corridor  which  has  to  be  used  by  all  UAVs  to  fly  to  the  area 
of  interest)  have  to  be  assigned  to  individual  team  members.  Secondly,  redundant  task  assignment  shall  be 
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avoided  in  order  to  maximise  team  effectiveness.  Thirdly,  team  members  shall  support  fellow  team 
members  by  carrying  out  tasks  that  facilitate  the  task  accomplishment  of  the  fellow  team  members.  The 
management  of  these  interdependencies  is  triggered  by  appropriate  desires. 

Communication  among  the  ACUs  is  structured  according  to  the  specific  dialogue  representation.  In  order 
to  keep  dialogues  running,  the  desire  ‘continue  dialogue’  becomes  active,  if  the  current  state  of  a  dialogue 
requires  an  ACU  to  send  a  message  next. 

Within  this  context,  various  environment  models  are  required.  First  of  all,  a  representation  of  actors  is 
needed,  which  is  instantiated  for  each  actor  in  the  setup,  here  for  each  ACU  piloting  a  UAV  and  for  the 
human  operator.  Closely  related  to  these  models  are  the  ones  which  describe  the  various  characteristics 
about  capabilities  and  resources  of  the  actors.  Since  the  actors  are  organised  in  a  team,  also  a  model  is 
necessary  about  how  a  team  is  to  be  defined.  This  model  is  for  example  attributed  with  the  members  of  the 
team  pointing  at  instantiations  of  actors.  With  respect  to  the  common  objective  of  a  team,  commitments 
have  to  be  represented.  These  are  further  described  by  the  task  which  is  related  to  the  commitment  and  the 
actor  or  team,  which  is  responsible  for  the  commitment.  As  commitments  refer  to  tasks,  tasks  have  to  be 
modelled.  Furthermore,  a  representation  of  dialogues,  their  states  and  transitions  are  needed,  which  again 
are  related  to  actors  and/or  teams  involved  in  an  interaction. 


4.4  Evaluation 

The  functionality  of  the  system  was  successfully  tested  in  a  simulation.  A  great  number  of  test  runs  were 
conducted.  Hereby,  different  scenarios  were  used,  some  of  which  considering  only  parts  of  the  overall 
setup  and  therefore  demanding  for  particular  key  capabilities  of  the  ACUs,  while  others  comprised  the 
complete  mission  scenario  in  order  to  test  the  overall  capabilities  of  the  ACUs. 

Figure  14  shows  the  course  of  a  simulation  test  run  with  a  reduced  scenario  considering  mission 
accomplishment  in  a  heterogeneous  UAV  team.  It  consisted  of  a  team  of  2  UAVs,  one  attack  UAV  (ID  4) 
and  another  one  (ID  0)  which  is  able  to  suppress  any  of  the  threats  being  around.  One  threat  area  covers 
the  location  of  the  high  value  target  to  be  attacked  as  shown  in  Figure  14  for  the  time  00:00. 

In  the  following,  both  UAVs  comply  with  their  commitments,  i.e.  the  attack  UAV  flies  towards  the  target, 
while  the  other  UAV  (ID  0)  starts  flying  towards  the  threat  (Figure  14,  02:07).  After  the  UAV  (ID  0)  has 
successfully  suppressed  the  threat  (Figure  14,  03:53),  it  informs  the  ACU  guiding  UAV  4  and  the  attack 
UAV  adjusts  its  flight  plan  and  continues  its  way  to  the  target,  now  being  escorted  by  the  other  UAV 
(Figure  14,  04:31).  After  having  successfully  attacked  the  target,  the  team  continues  the  request  dialogue 
initiated  by  the  operator  and  informs  him  about  the  successful  outcome  of  the  mission.  Finally,  both  UAVs 
start  returning  to  the  home  base  (Figure  14,  06:29  &  11:49). 

Within  the  project  described  here,  scenarios  consisting  of  up  to  10  known  and  unknown  threats  and  5 
UAVs  as  well  as  a  corridor  for  penetrating  into  the  action  theatre  have  successfully  been  tested  in  a 
simulated  environment  with  the  ACU  prototype  used  for  UAV  guidance  as  described. 

Besides  the  functionality  as  such  also  the  capability  of  the  prototype  to  behave  on  the  knowledge -based 
level  of  performance  was  successfully  evaluated  (see  [13]). 
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Figure  14:  Example  of  mission  accomplishment  of  a  team  of  2  UAVs  in  a  test  run 


4.5  Human-ACU  Teaming 

Finally,  this  work  also  provides  the  basis  for  human-machine  cooperation  although  its  main  focus  is  on 
machine-machine  cooperation  aspects.  This  add-on  could  be  achieved  because  of  a  consequent  provision 
of  models  of  goals  for  cooperation,  of  coordination  techniques,  and  of  dialogue  management  related 
knowledge.  Within  this  context,  an  experiment  was  conducted  in  order  to  gain  first  insight  to  problems 
arising  in  human-ACU  cooperation  and  to  derive  requirements  for  future  ACU  development.  There, 
human  pilots  had  to  control  up  to  three  UAVs  equipped  with  ACUs  to  accomplish  a  mission  as  described 
at  the  beginning  of  this  chapter  (see  [13][20]). 

For  the  experiments  the  general  hypothesis  was  stated  that  a  human  working  together  with  supporting 
ACUs  aboard  UAVs  is  able  to  handle  the  situation  at  large,  but  is  additionally  in  need  of  assisting 
functions.  Considering  the  fact,  that  the  ACUs  had  been  developed  as  cooperative  Supporting  ACUs,  the 
work  system  setup  shown  in  Figure  15  had  been  chosen  for  the  experiments. 
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Here,  a  pilot  of  a  manned  aircraft  forms  a  separate  work  system  and  gets  the  same  task  in  the  shape  of  a 
mission  order  as  the  UAV  team.  Although  not  being  part  of  the  same  work  system,  the  pilot  and  the  ACUs 
form  a  team  due  to  the  fact,  that  they  pursue  the  same  common  objective,  i.e.  the  mission.  Within  the 
achievement  of  this  common  objective,  the  coordination  of  their  activities  can  take  place  by  the  exchange 
of  messages. 


In  concrete  terms,  the  mission  objective  was  to  destroy  a  designated  target,  which  was  protected  by  several 
known  and  unknown  SAM  sites.  The  mission  tasks  derived  from  the  objective  were  to  attack  the  target 
and  to  suppress  or  destruct  enemy  air  defence.  Two  kinds  of  vehicles  were  involved,  namely  one  attack 
aircraft  being  capable  of  destroying  the  target  and  one  or  three  SEAD  aircraft  being  capable  of  suppressing 
or  destroying  SAM  sites.  Hereby,  several  configurations  were  considered: 

•  one  manned  attack  aircraft  and  one  SEAD  UAV, 

•  one  manned  attack  aircraft  and  three  SEAD  UAVs, 

•  one  attack  UAV  and  one  manned  SEAD  aircraft, 

•  one  attack  UAV  and  one  manned  and  two  unmanned  SEAD  aircraft. 

The  subjects  had  a  “teaming  GUI”  (see  Figure  16)  at  their  disposal  to  communicate  with  the  ACUs,  where 
they  could  generate  new  mission  tasks  such  as  “suppress  SAM-site  number  5”,  assign  these  to  either  a 
team  of  UAVs  or  a  single  UAV  and  adjust  or  abort  them.  Moreover,  information  about  e.g.  commitments 
of  team  members  was  provided  there. 

Additionally,  a  “navigation  GUI”  showed  the  tactical  situation,  so  that  the  subjects  could  monitor  the 
scenario  and  work  on  mission  tasks.  It  also  provided  necessary  information  to  the  subjects  concerning 
their  role  as  pilot  of  the  manned  aircraft.  It  could  be  controlled  by  autopilot  commands  (here:  speed  and 
heading  demands)  using  a  keyboard. 
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Figure  16:  Teaming  GUi  used  in  the  experiment 


Five  experienced  military  air  crew  members  in  the  age  from  31  to  50  served  as  subjects.  These  subjects 
conducted  several  missions  in  different  scenarios  for  each  of  the  configurations  listed  above.  Their 
workload  profile  was  evaluated  after  each  mission  using  the  NASA-TLX  method  [21].  Moreover,  they 
were  interviewed  to  be  able  to  state  problems  and  suggest  system  improvements. 

100%  T . 


56% 


Figure  17:  Average  workioad  of  subjects  in  different  configurations 


The  performance  of  the  human-machine  team  was  very  good.  More  than  90%  of  all  missions  could  be 
completed  successfully.  Regarding  the  workload  of  the  subjects,  it  was  on  a  medium  level  in  all 
configurations  that  have  been  investigated  (see  Figure  17  for  the  average  workload  of  humans  in  different 
configurations).  Interestingly,  there  is  only  a  small  increase  in  workload  when  changing  the  number  of 
UAVs  within  the  team  from  one  to  three.  Moreover,  the  increase  in  workload  when  changing  the  human 
role  from  attack  to  SEAD  is  as  large  as  when  adding  two  UAVs,  which  could  be  related  to  different 
performance  of  UAVs  in  the  SEAD  and  attack  role. 
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These  results  show  that  in  principle  human-ACU  cooperation  is  possible  with  the  approach  of  cognitive 
and  cooperative  automation  and  that  workload  is  on  a  medium  level  even  though  ACUs  were  not  designed 
to  work  together  with  humans.  Still,  further  improvement  potential  was  identified  in  the  following  areas 
after  having  analysed  interview  protocols: 

•  Team  Structure.  The  introduction  of  a  hierarchical  team  structure  is  encouraged,  although  team 
members  shall  be  capable  of  situation  dependent  deviation  from  leader  input. 

•  Abstraction  of  Interaction.  Interaction  with  ACUs  shall  be  as  abstract  as  possible  (e.g. 
specification  of  tasks  for  UAV  team  rather  than  detailed  instructions  for  single  UAVs).  In 
particular  it  shall  be  possible  to  give  instructions  on  different  abstraction  levels. 

•  Level  of  Automation.  ACUs  shall  be  able  to  act  on  a  high  level  of  automation  but  the  actual  level 
shall  be  adaptable  or  self-adapted  to  the  current  situation  and  task. 

•  Teamwork.  Cooperation  of  humans  and  ACUs  shall  be  based  on  a  common  agenda  and  anticipate 
future  actions  of  team  members. 

•  Task  completion.  The  capability  of  ACUs  to  actually  accomplish  tasks  has  to  be  improved. 

•  Communication  within  team.  Vocabulary  shall  be  adapted  for  more  intuitive  understanding  of 
humans.  Moreover,  the  number  of  dialogs  and  messages  shall  be  reduced  to  a  minimum  in  order 
to  avoid  overload  of  humans. 

•  Assistant  System.  The  human  team  member  shall  be  supported  by  an  electronic  assistant  providing 
him  or  her  with  situation  awareness  concerning  team  members,  task  assignment  and  information 
flow  within  team  as  well  as  accomplishment  of  his  or  her  primary  mission  related  task,  i.e.  aircraft 
guidance. 

Ongoing  work  is  intended  to  account  for  these  findings  and  bring  together  the  different  aspects  of  human- 
machine  and  machine-machine  cognitive  cooperation  in  manned-unmanned  multi-agent  scenarios.  In  the 
field  of  supervisory  control  of  UAVs  the  question  of  how  to  reduce  the  operator-to-vehicle  ratio  will  be 
predominant.  Hereby,  cognitive  and  cooperative  automation  offer  solutions  for  many  of  the  upcoming 
questions. 


5.0  CONCLUSIONS 

Our  research  work  on  cognitive  and  cooperative  automation  in  military  aerial  applications  is  driven  by  the 
goal  to  enable  a  pilot  or  UAV  operator  to  eventually  guide  more  than  one  aircraft  at  the  same  time. 
Undoubtedly,  the  workload  in  suchlike  work  situations  can  easily  exceed  certain  unwanted  limits 
especially  when  anticipating  a  dense  and  dynamic  wartime  theatre.  However,  current  military  scenarios 
require  the  close  cooperation  between  manned  and  unmanned  assets.  This  entails  even  airborne  UAV 
guidance  and  an  increasing  vehicle  to  operator  ratio.  Nevertheless,  from  a  pilot’s  perspective  it  is 
undisputable  that  operating  multiple  aircraft  with  today  available  guidance  and  control  technologies  would 
be  simply  not  worth  any  discussion.  The  guidance  concept  based  on  the  approach  of  cognitive  and 
cooperative  automation  includes  functional  aspects  like 

•  the  availability  of  control  strategies  using  higher  level  tasks  assignments  instead  the  usage  of 
parameters  for  action  control, 

•  the  assumption  of  goal  oriented  rational  behaviours  following  these  tasks  including  the 
capability  of  a  reasonable  self-assignment  of  tasks  and  the  making  of  local  decisions  in  case  of 
disrupted  communication  or  situational  circumstances  beyond  the  perception  of  the  operator,  and 
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•  the  existence  of  a  cooperative  work  relationship  directed  towards  common  objectives  between 
the  operator  and  the  controlled  aircraft  and  even  between  the  unmanned  aircraft  if  needed. 

Involving  artificial  cognition  into  the  work  system  design  based  on  a  better  insight  on  human  cognition 
mechanisms,  as  we  are  utilising  in  our  approach,  leads  to  further  opportunities  with  respect  to  human- 
machine  system  performance  enhancements.  The  required  functions  of  artificial  components  supporting 
the  human  operator,  in  particular  artificial  cognition,  can  be  specified  on  a  more  rational  basis,  because  it 
is  better  known  where  the  human  operator  really  needs  support.  Effective  cooperation  on  a  cognitive  level 
between  artificial  systems  (automation)  and  the  human  operator  can  be  made  a  design  characteristic.  This 
can  be  achieved  by  use  of  quantitative  models  of  human  behaviour  like  those  we  have  in  mind  ourselves 
when  we  are  cooperating  in  a  team  e.g.  models  of  operator  workload  in  certain  work  situations  or  models 
concerning  available  human  resources.  The  use  of  the  knowledge  about  the  capabilities  and  the 
architecture  of  human  cognition  as  a  reference  is  an  eligible  way  to  achieve  better  architectures  for 
artificial  cognitive  systems. 

In  fact,  it  is  not  an  easy  task  to  implement  a  system  involving  artificial  cognition  as  claimed  by  our  paper. 
However,  there  are  already  several  field  implementations  on  their  way.  But  non  of  these,  though,  has  yet 
reached  that  high  level  of  flexibility,  interoperability,  and  autonomy  accompanied  with  a  maximum 
performance  in  a  sense  of  human  factors  and  human-automation  integration  characteristics  that  military 
user  and  operators  require  and  deserve.  We  hope  to  be  substantially  contributing  to  the  achievement  of  this 
goal  by  our  interdisciplinary  and  embracing  approach,  being  aware  that  many  unsolved  problems  still  have 
to  be  tackled. 

As  to  our  belief,  one  of  the  most  relevant  fields  of  future  research  will  be  associated  with  the  term  of 
adaptive  autonomy.  Our  experimental  findings  clearly  indicate  that  there  is  a  strong  need  for  self 
adaptation  of  the  level  of  autonomy  of  the  UAVs.  Operators  will  demand  to  interact  with  the  UAVs  on 
various  abstraction  levels  ranging  from  dedicated  parameter  settings  over  task  based  guidance  to  team 
capability  management.  Given  that,  combined  with  appropriate  interfaces  (e.g.  speech)  will  open  the  door 
to  human-automation  cooperation  akin  to  cooperation  in  purely  human  teams. 
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